Effect of Cost Function Complexity and Dimensionality on Newton’s Method Convergence Rate

Today, Newton’s Method is commonly used in machine learning and optimization to locate local minima of cost functions. Because it is such a general tool, the cost functions that is implemented on can range widely in terms of both their dimensionality and their complexity. However, little is understood about how Newton’s Method performs on cost functions with different dimensionalities and complexities. In the simplest case, where the cost function is a paraboloid, one can show that Newton’s Method converges in a single iteration to the minimum. We can increase the complexity of our cost functions by considering perturbative deviations from the paraboloid cost function. Intuitively, one should expect that as we increase the cost function complexity, the convergence rate will decline. However, as of now, there is no simple way to parametrize the complexity of a cost function, and thus, little is known about the way that cost function complexity affects the convergence of Newton’s Method. In this paper, I introduce a method of generating perturbed cost functions which allow us to systematically deviate from the simple paraboloid case. These perturbed cost functions are parametrized by a parameter N , which describes the number density of gaussian pertubation sites and is thus related to the complexity of the generated cost functions. By empirically fitting a model to convergence rate data, I find that the convergence rate depends strongly on both N and D (the dimensionality of the cost function). This dependence on N and D leads me to the hypothesis that the convergence rate of Newton’s Method depends directly on the mean spacing between pertubations.